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Some education programs’ early positive effects disappear over time. Other pro- 
grams have unanticipated positive long-term effects. Foundations warn of the 


dangers of putting too much weight on in-program effects, which “often fade 
over time.” The U.S. Department of Education’s Institute of Education Sciences 
(IES) even has a special funding category dedicated to continued follow-up to 


explore these issues. 


This Issue Focus tackles the topic of post-program effects in postsecondary ed- 
ucation, a previously unexamined context. Are in-program effects—that is, the 
effects observed while the program was active—maintained once the program 
ends? Do they grow and improve? Or do they fade out? 


This investigation capitalizes on two decades of rigorous program evaluations 
conducted by MDRC, including approximately 25 postsecondary programs, 
to consider these questions. The programs varied widely in terms of their fea- 
tures (like financial support, advising, learning communities, tutoring, success 
courses, instructional reforms, and communication campaigns), their duration 
(lasting from one semester to three years), the populations they served, and their 
contexts. Each evaluation used a well-executed randomized controlled trial 
(RCT) design to estimate program effectiveness. RCTs are often considered the 
gold standard for evaluating program effectiveness. 


‘The results are striking: During the year after these programs ended, effects on 
academic progress (as measured by credit accumulation, an indicator of prog- 
ress toward a degree) were consistently maintained. There is no evidence of dis- 
cernible fade-out after these programs ended, providing encouraging informa- 
tion about the lasting value of many postsecondary programs, as well as the 
value of evidence collected from ongoing programs. There was also no evidence 
of post-program growth, perhaps prompting the need for the development of 
programs that equip students for success beyond their current durations. 


HYPOTHESES REGARDING POST-PROGRAM EFFECT 
TRAJECTORIES 


Theorists from various education-related fields have hypothesized what may 
happen to effects after a program ends. The most prominent set of hypotheses 
comes from early education researchers, who outline three processes by which 


program effects may be sustained, grow, or fade over time: development of tri- 
fecta skills, “foot-in-the-door” programs, and sustaining environments. Though 
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initially developed around a different transition point, these theories may also apply to postsecondary post-program 
effects. Trifecta skills, in a college-level context, may be fostered by something like a student success course that 
teaches study and time management skills as students begin their studies. Those skills are “malleable, fundamental, 
and would not have developed eventually in the absence of the program.” Foot-in-the-door programs, such as pro- 
viding emergency financial aid, are often helpful during sensitive periods in students’ lives to “avoid imminent risks” 
to dropping out of school or “seize emerging opportunities,” to continue postsecondary education. Finally, sustaining 
environments, like supported entry into a high-quality college or university after a summer bridge program, help 
students “into high-quality environments that support their continued growth.” 


MAINTENANCE OF EFFECTS AFTER POSTSECONDARY PROGRAMS END 


This investigation, examining what happens to program effects after a postsecondary education program ends, re- 
lies on a unique database, known as The Higher Education Randomized Controlled Trials (THE-RCT), explained 
in Box 1. 


Box 1 


The Higher Education Randomized Controlled Trials (THE-RCT) 


The cornerstone of THE-RCT is a restricted access file (RAF) containing de-identified student-level data from 
31 of MDRC’s higher education randomized controlled trials (RCTs), involving 45 institutions and 67,400 stu- 
dents. Data include demographics (such as gender and race and ethnicity), outcomes (such as enrollment, credits 
earned, and credentials), and study-related variables. The programs range from “light touch” interventions to 
comprehensive interventions, with varying durations (one semester to three years), targeting various popula- 
tions and contexts. Several of the studies had multiple intervention arms, meaning the RAF includes 41 unique 
programs. 


How to access the RAF: The data are available to qualified researchers who have obtained access through the 
University of Michigan’s ICPSR. 


THE-RCT also includes several free, publicly available documents: 


THE-RCT: RAF User's Guide—describes the data, processing rules, and variable naming conventions. 


THE-RCT: RAF Codebook—includes a list of the RAF variables, variable definitions, variable availability by 
RCT, and timing of outcome data availability. 


THE-RCT: RCT-level Database—provides a list of the RCTs in the RAF, summary information about each 
RCT, and a list of the colleges that participated in the RCTs. 


How was the RAF used in this Issue Focus: This Issue Focus capitalizes on nearly all the RCTs in the RAF and 
four RCTs not in the RAF (because RCTs conducted at the City University of New York are not currently part of 
the publicly accessible RAF). In total, six RCTs were excluded because the follow-up did not extend post-program 
or no comparable outcome measure existed. Also, two RCTs had multiple intervention arms that were analyzed 
as separate programs. The analyses may be replicated using SAS software (code available upon request) and THE- 
RCT: RAF. 


WHAT HAPPENS AFTER THE PROGRAM ENDS? 


Figure 1 presents findings from each of 29 unique programs represented in the database for THE-RCT. The y-axis 
represents the estimated effect of the program on cumulative credits earned, an indicator of progress toward a degree 
measured by nearly all of the studies in the database. These effects refer to the average number of additional credits 
students earned because of the program—credits they would not have earned in the absence of the program. The 
x-axis represents time in years, centered around the final program semester. Negative time values (zero included) are 
“in-program” semesters. Positive time values are “post-program” semesters, the emphasis of this Issue Focus. Since 
most studies have at least one year of post-program follow-up, that time frame is highlighted—that is, the time be- 
tween the two vertical dashed lines. 


Figure 1. Effects on Credits Earned over Time Across RCTs of 29 Programs 
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Source: MDRC calculations using data from John Diamond, Michael J. Weiss, Colin Hill, Austin Slaughter, and Stanley Dai, “The 
Higher Education Randomized Controlled Trial (THE-RCT)” (Ann Arbor, MI: Inter-university Consortium for Political and Social 
Research, 2003-2019). Accessed 2021-03-10. https://doi.org/10.3886/ICPSR37932.v1. 


Note: The horizontal dashed red line highlights an evaluation of the Opening Doors program, described in the text. 


To determine the extent that program effects are maintained, grow, or fade out after the program ends, it is helpful to 
start by looking at the estimated effect at the end of the program (that is, the vertical line at time = 0) and compare it 
to the estimated effect at a later post-program time point. If the two values are similar, effects were maintained, while 
large increases represent evidence of growth and large decreases represent evidence of loss or “fade-out.” 


For example, the red dashed line in Figure 1 highlights an evaluation of the Opening Doors program in Ohio. This 
program offered students from low-income families attending Lorain County Community College and Owens Com- 
munity College enhanced advising services and a modest stipend for two semesters. At the end of two semesters, stu- 
dents who were offered the enhanced advising and stipend had a 0.46 credit advantage over their control group coun- 
terparts who were not offered these additional services. One year after the program ended, that advantage remained 
similar—students who were offered the program were a total of 0.58 credits ahead of their control group counterparts. 
The +0.12 increase in credits earned is neither substantively meaningful nor statistically significant, implying that the 
effects observed during the program period were maintained one year after the program ended. 


WHAT HAPPENS AFTER THE PROGRAM ENDS? 


The post-program findings from the Opening Door’s enhanced advising evaluation exemplify the overall pattern 
of findings across the programs examined—on average, across all programs with at least one year of post-program 
follow-up, the effect on cumulative credits earned at the end of the program remained about the same one year later. 
Specifically, when pooled across all studies, the change in the effect on cumulative credits earned during the first 
post-program year is just +0.02 credits. This change is neither substantively meaningful nor statistically significant. 
This means that, for the typical program, however many additional credits students earned during the program, be- 
cause of the program, this number was basically the same one year later. At the two-year mark (for those studies where 
data are available), results were largely the same. 


The overall finding that effects on credits earned were maintained one-year post-program could, in theory, mask 
differences among programs. As noted above, the studied programs varied widely in terms of their in-program effec- 
tiveness, their features, their duration, the populations they served, and their contexts—factors that could influence 
what happens after the programs’ end. Surprisingly, despite these many differences, in practice, the evidence points 
to a consistent pattern of post-program maintenance of effects in every study.' There simply is not clear evidence of 
post-program growth—that is, improved effects—or fade-out, for any of the programs examined. 


CONCLUSION 


‘The finding that effects on credit accumulation are broadly maintained after postsecondary programs end should en- 
courage education reformers concerned about fade-out. While in-program effects are sometimes important on their 
own, benefits maintained into the future are especially powerful. 


Some important points are worth noting: 


First, the above analyses focus on a single outcome—cumulative credits earned. While that outcome is an important 
indicator of academic progress, it is not the only important one targeted by postsecondary programs. Exploring other 
outcomes, like enrollment, grade point average, or degree completion, is an area ripe for future research. 


Second, while post-program maintenance of effects is better than fade-out, growth would be even better. This Issue 
Focus begins the conversation—did the programs included in the analyses presented here intend to affect skills that 
should continue to serve students after the program period ended? Were the components designed to affect such skills 
well-implemented? If so, program developers should consider whether they need to further bolster those skills, or 
address other skills, to achieve post-program growth. Are there programs with rigorous evidence of effects that grow 
after a program ends? If so, what program features help foster that growth? 


This Issue Focus is the first in a series of papers that are part of THE-RCT project. Other papers will examine: 


= Which program components (for example, enhanced advising or increased financial support) are associated with 
larger improvements in student success? 


= What magnitude of improvements in student success can various higher education programs reasonably expect 
to achieve? 


= How well do short-term effects predict long-term effects? 


1 A test for variation in effects across programs yields (24) = 16.0; p = 0.888. 


WHAT HAPPENS AFTER THE PROGRAM ENDS? 


Like the analyses presented here, these questions will be examined using an exciting new resource available to post- 
secondary researchers—a restricted access file with de-identified, student-level data from 26 out of the 31 postsecond- 
ary RCTs that MDRC has conducted since 2001, as shown in Box 1. Ultimately, MDRC hopes that this database will 
inspire new scholarship on these and other topics that will help improve outcomes for low-income, underrepresented, 
and underprepared students, who have long been a focus of MDRC’s higher education studies. To learn more about 
The Higher Education Randomized Controlled Trials (THE-RCT) database and how to access it, visit https://www. 
icpsr.umich.edu/web/ICPSR/studies/37932 or contact michael.weiss@mdrc.org. 
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